Papers with training process
The Pragmatic Mind of Machines: Tracing the Emergence of Pragmatic Competence in Large Language Models (2026.eacl-long)
Copied to clipboard
| Challenge: | Current large language models (LLMs) have demonstrated emerging capabilities in social intelligence tasks, including implicature resolution and theory-of-mind reasoning. |
| Approach: | They introduce a dataset grounded in the pragmatic concept of alternatives to evaluate whether large language models can accurately infer nuanced speaker intentions. |
| Outcome: | The proposed model can infer nuanced speaker intentions by inferring the speaker’s intended meaning and explaining when and why a speaker would choose one utterance over its alternative. |
Measuring and Improving Chain-of-Thought Reasoning in Vision-Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | Vision-language models have demonstrated strong efficacy as visual assistants . however, evaluation of their reasoning capabilities requires a costly benchmark . |
| Approach: | They propose a pipeline to measure the reasoning consistency of vision-language models . they propose supervised fine-tuning of VLMs and feedback from LLMs . |
| Outcome: | The proposed framework reduces cost while ensuring the generation of a high-quality dataset. |
Data Augmentation by Data Noising for Open-vocabulary Slots in Spoken Language Understanding (N19-3)
Copied to clipboard
| Challenge: | Neural networks are used to understand spoken language understanding (SLU) but it is difficult to recognize the slots of unknown words or ‘open-vocabulary’ slots because of the high cost of creating a manually tagged SLU dataset. |
| Approach: | They propose to use a recurrent neural network to nois slots for data augmentation by using an attention-based bi-directional recurrence neural network. |
| Outcome: | The proposed method achieves performance improvements of up to 0.57% and 3.25 in intent prediction (accuracy) and slot filling (f1-score) and 0.53% accuracy. |
Improving Privacy Guarantee and Efficiency of Latent Dirichlet Allocation Model Training Under Differential Privacy (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Latent Dirichlet allocation (LDA) is a widely used topic model to discover the latent semantic of text data. |
| Approach: | They propose to combine a subsampling method with CGS to improve efficiency while amplifying privacy by using a novel metric, the efficiency–privacy function. |
| Outcome: | The proposed algorithm improves efficiency while amplifying privacy while subsampling in CGS increases efficiency while preserving privacy. |
Improving Mongolian-Chinese Neural Machine Translation with Morphological Noise (P19-2)
Copied to clipboard
| Challenge: | Existing models for Mongolian-Chinese translation are based on recurrent, convolutional neural networks or completely eliminate recurrence connections. |
| Approach: | They propose a adversarial training model to alleviate the UNK problem in Mongolian-Chinese machine translation by adding a screener to the model to emphasize the added Mongolian morphological noise. |
| Outcome: | The proposed model reduces training time and improves accuracy in Mongolian-Chinese translation tasks. |
Adaptively Scheduled Multitask Learning: The Case of Low-Resource Neural Machine Translation (D19-56)
Copied to clipboard
| Challenge: | Neural Machine Translation suffers from the lack of bilingual data in low-resource scenarios. |
| Approach: | They propose to inject inductive biases into Neural Machine Translation (NMT) using auxiliary syntactic and semantic tasks. |
| Outcome: | The proposed approach improves translation quality by reweighing training data of main and auxiliary tasks based on their contributions to generalisability of main task. |
PRILoRA: Pruned and Rank-Increasing Low-Rank Adaptation (2024.findings-eacl)
Copied to clipboard
| Challenge: | Several approaches to parameter-efficient fine-tuning have been proposed . low-rank Adaptation (LoRA) does not consider the varying importance of each layer . |
| Approach: | They propose a method that allocates a different rank for each layer and performs pruning throughout the training process. |
| Outcome: | The proposed method is based on eight GLUE benchmarks and is currently the state of the art. |
Estimating Soft Labels for Out-of-Domain Intent Detection (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to detect out-of-dominance (OOD) intents are limited by the lack of OOD samples. |
| Approach: | They propose an adaptive soft pseudo labeling method that can estimate soft labels for pseudo OOD samples when training OOD detectors. |
| Outcome: | The proposed method outperforms competing methods on three benchmark datasets and consistently outperformed previous methods. |
Adaptor: Objective-Centric Adaptation Framework for Language Models (2022.acl-demo)
Copied to clipboard
| Challenge: | Adaptor library aims to simplify complex training processes requiring customizations. |
| Approach: | They introduce Adaptor library which transposes traditional model-centric approach to objective-centric training pipeline with Objective as central abstraction. |
| Outcome: | The proposed framework simplifies training processes and improves reproducibility. |
Enhancing Self-Attention via Knowledge Fusion: Deriving Sentiment Lexical Attention from Semantic-Polarity Scores (2024.starsem-1)
Copied to clipboard
| Challenge: | Existing methods to inject lexical features into self-attention mechanisms have shown remarkable performance across various downstream tasks in NLP. |
| Approach: | They propose to inject lexical features into the self-attention mechanism of Transformer-based models by injecting lexicon-based Sentiment Lexical Attention into the attention scores throughout the training process. |
| Outcome: | The proposed method shows significant performance improvements on the NSMC sentiment classification benchmark and is able to perform in out-of-domain tasks. |
Active Learning Design Choices for NER with Transformers (2024.lrec-main)
Copied to clipboard
Robert Vacareanu, Enrique Noriega-Atala, Gus Hahn-Powell, Marco A. Valenzuela-Escarcega, Mihai Surdeanu
| Challenge: | In the field of natural language processing, active learning is a technique that is used to decide which examples are worth annotating . a number of studies have focused on sequence classification, text classification, question answering, and question answering. |
| Approach: | They propose two different approaches to deal with partially-annotated sentences . they propose an annotation scheme that can be used to train with tokens . |
| Outcome: | The proposed approaches achieve comparable or better performance than sentence-level annotations with a smaller number of annotated tokens. |
Multilingual Agreement for Multilingual Neural Machine Translation (2021.acl-short)
Copied to clipboard
| Challenge: | Existing models that only use auxiliary languages to encourage multilingual agreement ignore the relationships between different language pairs. |
| Approach: | They propose a multilingual agreement-based method which explicitly models the agreement between different translation directions by randomly substituting some fragments of the source language with their counterpart translations of auxiliary languages. |
| Outcome: | The proposed method improves on the multilingual translation task of 10 language pairs. |
LoRA-MGPO: Mitigating Double Descent in Low-Rank Adaptation via Momentum-Guided Perturbation Optimization (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Low-Rank Adaptation (LoRA) adapts large language models by training only a small fraction of parameters, but as the rank of the low-rank matrices increases, LoRA exhibits an unstable “double descent” phenomenon, which delays convergence and impairs generalization by causing instability due to the attraction to sharp local minima. |
| Approach: | They propose a framework that incorporates Momentum-Guided Perturbation Optimization (MGPO) MGPO stabilizes training dynamics by mitigating double descent phenomenon and guiding weight perturbations using momentum vectors from the optimizer’s state. |
| Outcome: | The proposed framework improves performance on natural language understanding benchmarks and shows that it improves convergence and generalization. |
Combining Weakly Supervised ML Techniques for Low-Resource NLU (2021.naacl-industry)
Copied to clipboard
| Challenge: | Recent advances in transfer learning have improved the performance of virtual assistants . however, meager training data is often a key bottleneck in creating voice-enabled applications . |
| Approach: | They propose to use unsupervised and semi-supervised techniques to improve NLU accuracy . they incorporate anonymized, unlabeled and automatically transcribed user utterances into training . |
| Outcome: | The proposed methods improve NLU accuracy in low-resource settings by integrating unsupervised and SSL techniques. |
Building a Family of Data Augmentation Models for Low-cost LLM Fine-tuning on the Cloud (2025.coling-industry)
Copied to clipboard
| Challenge: | Existing models for learning large language models are expensive and difficult to build and fine-tune. |
| Approach: | They propose a family of data augmentation models to improve model fine-tuning efficiency . they leverage powerful LLMs to expand, refine and re-write instructions and responses . |
| Outcome: | The proposed models improve the efficiency of model fine-tuning by leveraging small datasets and quality assessment techniques. |
TernaryBERT: Distillation-aware Ultra-low Bit BERT (2020.emnlp-main)
Copied to clipboard
| Challenge: | Transformer-based pre-training models like BERT are computationally expensive and limited to resource-constrained devices. |
| Approach: | They propose a method which ternarizes the weights in a fine-tuned BERT model. |
| Outcome: | The proposed method outperforms the other methods on the GLUE and SQUAD benchmarks while being 14.9x smaller. |
Checkpoint Reranking: An Approach to Select Better Hypothesis for Neural Machine Translation Systems (2020.acl-srw)
Copied to clipboard
| Challenge: | Neural Machine Translation (NMT) has produced excellent results in the field of machine translation due to generation of high-quality translations for different language pairs. |
| Approach: | They propose a method of re-ranking the outputs of Neural Machine Translation systems by focusing on the decoder's ability to generate distinct tokens and without the use of any language model or data. |
| Outcome: | The proposed method achieves translation improvement up to +0.16 BLEU points over baseline. |
Synonym-unaware Fast Adversarial Training against Textual Adversarial Attacks (2025.findings-naacl)
Copied to clipboard
| Challenge: | Existing adversarial defense methods rely on predetermined linguistic knowledge and assume that attackers’ synonym candidates are known, which is often unrealistic. |
| Approach: | They propose a Fast Adversarial Training method that leverages single-step perturbation generation and effective perturbation initialization to improve model robustness without requiring synonym awareness. |
| Outcome: | Experiments show that the proposed method outperforms existing models under character-level and word-level attacks while still maintaining the correct syntax. |
Parallel Continuous Chain-of-Thought with Jacobi Iteration (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to continuous CoT rely on sequential decoding of latent thought tokens, which leads to long training time and low inference speed. |
| Approach: | They propose a parallel continuous chain-of-thought which updates latent thought tokens iteratively in parallel instead of sequentially and improves both training and inference efficiency. |
| Outcome: | The proposed method saves 50% of training and inference time while maintaining stability and robustness in training. |
CDL: Curriculum Dual Learning for Emotion-Controllable Response Generation (2020.acl-main)
Copied to clipboard
| Challenge: | Existing methods for generating emotion-controllable response are inadequate due to content consistency and lack of coherence. |
| Approach: | They propose a framework that extends the emotion-controllable response generation to a dual task to generate emotional responses and emotional queries alternatively. |
| Outcome: | The proposed framework outperforms baseline models in coherence, diversity, and relation to emotion factors. |
ELO: Efficient Layer-Specific Optimization for Continual Pretraining of Multilingual LLMs (2026.eacl-industry)
Copied to clipboard
Hangyeol Yoo, ChangSu Choi, Minjun Kim, Seohyun Song, SeungWoo Song, Inho Won, Jongyoul Park, Cheoneum Park, KyungTae Lim
| Challenge: | Recent studies have focused on enhancing multilingual large language models (MLLMs) for specific languages. |
| Approach: | They propose an efficient layer-specific optimization method to enhance continual pretraining (CP) for specific languages in multilingual large language models (MLLMs). |
| Outcome: | The proposed method achieves a training speedup of up to 6.46 times compared to existing methods while improving target language performance by up to 5.2% on qualitative benchmarks. |
Towards Generative Aspect-Based Sentiment Analysis (2021.acl-short)
Copied to clipboard
| Challenge: | Existing work on Aspect-based sentiment analysis ignores the rich label semantics of ABSA. |
| Approach: | They propose to tackle various ABSA tasks in a unified generative framework . they propose to use annotation-style and extraction-style modeling to enable training . |
| Outcome: | The proposed framework achieves state-of-the-art on four ABSA tasks across multiple benchmark datasets. |
The Role of Data Curation in Image Captioning (2024.eacl-long)
Copied to clipboard
| Challenge: | Existing image captioning models treat all samples equally, neglecting mismatched data . Several other techniques have relied on curriculum learning strategies to adapt learning to the difficulty of the task. |
| Approach: | They propose to actively curate difficult samples in datasets using curriculum learning strategies to improve captioning models. |
| Outcome: | The proposed methods outperform existing models on the Flickr30K and COCO datasets. |
Provably Confidential Language Modelling (2022.naacl-main)
Copied to clipboard
| Challenge: | Existing methods to train language models without memorizing sensitive data are mismatched and can be difficult to screen and filter. |
| Approach: | They propose a method to train language generation models while protecting the confidential segments of training data. |
| Outcome: | The proposed method prevents unintended memorization by randomizing parts of the training process while protecting strong confidentiality. |
Self-Training with Differentiable Teacher (2022.findings-naacl)
Copied to clipboard
| Challenge: | Existing methods for self-training are interpreted as teacher-student frameworks, where the teacher generates pseudo-labels and the student makes predictions. |
| Approach: | They propose a differentiable self-training method that treats teacher-student as a Stackelberg game where a leader is always in a more advantageous position than a follower. |
| Outcome: | The proposed model outperforms existing methods on semi- and weakly-supervised learning tasks on semi and weak supervised tasks. |
A Close Look into the Calibration of Pre-trained Language Models (2023.acl-long)
Copied to clipboard
| Challenge: | Pre-trained language models (PLMs) may fail in giving reliable estimates of their predictive uncertainty. |
| Approach: | They conduct fine-grained control experiments to study the dynamic change in PLMs’ calibration performance in training. |
| Outcome: | The proposed methods significantly reduce PLMs’ confidence in wrong predictions. |
VEG: Verbal 𝜖-greedy for Semantic Exploration in Multi-Turn RL Agents (2026.acl-industry)
Copied to clipboard
Yongchang Hao, Jie Hao, Yongsheng Mei, Ze Ye, Junyi Chai, Bin Guo, Benjamin Z. Yao, Chenlei Guo, Lili Mou
| Challenge: | Standard RL approaches suffer from reward sparsity and mode-seeking behavior . lack of diversity hinders exploration necessary for optimal learning . |
| Approach: | They propose a framework that leverages external feedback as a dynamic control variable to explicitly balance exploration and exploitation within the semantic space. |
| Outcome: | Experiments on Tau Bench and SearchQA show that the proposed framework outperforms standard RL baselines. |
Multilingual Neural Machine Translation with Language Clustering (D19-1)
Copied to clipboard
| Challenge: | Existing work on multilingual neural machine translation has been neglected due to its burdensome training process. |
| Approach: | They develop a framework that clusters languages into different groups and trains one multilingual model for each cluster. |
| Outcome: | The proposed model reduces the cost of training and improves translation accuracy. |
Analyzing the Source and Target Contributions to Predictions in Neural Machine Translation (2021.acl-long)
Copied to clipboard
| Challenge: | Neural Machine Translation models are influenced by two types of context, source and target, but none explicitly evaluates relative contribution to generation decision. |
| Approach: | They propose to adopt a variant of Layerwise Relevance Propagation which evaluates relative contributions to the generation decision by a proportion of token influence. |
| Outcome: | The proposed model can evaluate the relative contribution of source and target to the generation decision by using a variant of Layerwise Relevance Propagation (LRP) |
Learning to Ground Visual Objects for Visual Dialog (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to ground visual objects are inadequate for visual dialog . a posterior distribution is inferred from context and questions, while posterior distributions are used to facilitate visual objects grounding. |
| Approach: | They propose a method to learn to ground visual objects for visual dialog using prior and posterior distributions over visual objects to facilitate visual objects grounding. |
| Outcome: | The proposed approach improves the existing models in generative and discriminative settings by a significant margin. |
M3: A Multi-View Fusion and Multi-Decoding Network for Multi-Document Reading Comprehension (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for multi-document reading comprehension cannot make full of the advantages of both approaches. |
| Approach: | They propose a multi-view fusion and multi-decoding method that integrates multiple documents for answering questions. |
| Outcome: | The proposed method improves on two mainstream multi-document reading comprehension datasets. |
Coherence-Aware Neural Topic Modeling (D18-1)
Copied to clipboard
| Challenge: | Topic models are evaluated for their ability to describe documents well (i.e. low perplexity) topic coherence is not optimized for and is only evaluated after training. |
| Approach: | They propose to incorporate a topic coherence objective into the training process by incorporating a coherency objective into a model. |
| Outcome: | The proposed model exhibits similar level of perplexity as baseline models but significantly higher topic coherence. |
Building Persona Consistent Dialogue Agents with Offline Reinforcement Learning (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to improve persona consistency are centered around supervised learning or online reinforcement learning (RL). Existing approaches to improve consistency are expensive and require additional training. |
| Approach: | They propose an offline supervised learning framework to improve persona consistency of dialogue systems by punishing and rewarding specific utterances. |
| Outcome: | The proposed framework improves both the persona consistency and dialogue quality of a state-of-the-art social chatbot. |
LLäMmlein: Transparent, Compact and Competitive German-Only Language Models from Scratch (2025.acl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have achieved remarkable success, yet this progress is predominantly centered on English. |
| Approach: | They create two German-only decoder models from scratch and publish them for the (German) NLP research community to use. |
| Outcome: | The two models performed competitively on the German SuperGLEBer benchmark, but performance improvements plateaued early during training, offering valuable insights into resource allocation for future models. |
Annotations on a Budget: Leveraging Geo-Data Similarity to Balance Model Performance and Annotation Cost (2024.lrec-main)
Copied to clipboard
| Challenge: | Current foundation models have shown impressive performance across various tasks, but they are not effective for everyone due to the imbalanced geographical and economic representation of the data used in the training process. |
| Approach: | They propose to identify the data to be annotated to balance model performance and annotation costs by finding countries with visual similarity for the topics. |
| Outcome: | The proposed methods improve model performance and reduce annotation costs by using data from countries with higher visual similarity for these topics. |
Incorporating Latent Meanings of Morphological Compositions to Enhance Word Embeddings (P18-1)
Copied to clipboard
| Challenge: | Existing word embedding methods learn semantic information at word level while neglecting meaningful inner structures of words like morphemes. |
| Approach: | They propose to use latent meanings of morphological compositions of words to train word embeddings. |
| Outcome: | The proposed models outperform baseline models on word similarity, syntactic analogy and text classification tasks. |
Encouraging Good Processes Without the Need for Good Answers: Reinforcement Learning for LLM Agent Planning (2025.emnlp-industry)
Copied to clipboard
| Challenge: | Currently, the dominant end-to-end reinforcement learning paradigm for agents in Large Language Models (LLMs) employs multi-objective optimization that jointly trains both planning and answer summarization capabilities. |
| Approach: | They propose a framework that decouples the training process to enable a focused, single-objective optimization of the planning module. |
| Outcome: | The proposed framework achieves an 8%–12% improvement in planning performance compared to end-to-end baselines. |
Improving Model Generalization: A Chinese Named Entity Recognition Case Study (2021.acl-short)
Copied to clipboard
| Challenge: | Named Entity Recognition (NER) is a fundamental building block for various downstream natural language processing tasks due to the ambiguous word boundaries and complex composition. |
| Approach: | They propose to resample entities within the same category to encourage a model to leverage both name and context knowledge in the training process. |
| Outcome: | The proposed method significantly improves a model’s ability to detect unseen entities, especially for company, organization and position categories. |
Faithful and Plausible Natural Language Explanations for Image Classification: A Pipeline Approach (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing explanation methods for image classification struggle to provide faithful and plausible explanations for predictions. |
| Approach: | They propose a natural language explanation method that can be applied to any CNN-based classifier without altering its training process or affecting predictive performance. |
| Outcome: | The proposed method can be applied to any CNN-based classifier without altering its training process or affecting predictive performance. |
Exploring and Adapting Chinese GPT to Pinyin Input Method (2022.acl-long)
Copied to clipboard
| Challenge: | a frozen GPT can generate state-of-the-art performance on perfect pinyin, but performance drops when input includes abbreviated pinyan, which links to even larger number of Chinese characters. |
| Approach: | They propose to use Chinese GPT to generate fluent sentences using abbreviated pinyin. |
| Outcome: | The proposed approach improves on abbreviated pinyin across all domains. |
Signer Diversity-driven Data Augmentation for Signer-Independent Sign Language Translation (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing methods for sign language translation (SLT) rely on signer identity labels, which is often impractical and costly in real-world applications. |
| Approach: | They propose a signer diversity-driven data augmentation method that can generalize to signers not encountered during training. |
| Outcome: | The proposed method achieves state-of-the-art results without relying on signer identity labels. |
Improving Dialogue State Tracking with Turn-based Loss Function and Sequential Data Augmentation (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing models rely on a traditional cross-entropy loss function during training, which may not be optimal for improving the joint goal accuracy. |
| Approach: | They propose a Turn-based Loss Function that penalises the model if it inaccurately predicts a slot value at the early turns more so than in later turns to improve joint goal accuracy. |
| Outcome: | The proposed techniques improve the state-of-the-art model by approximately 7-8% relative reduction in error and achieve a new state- of-the art joint goal accuracy with 59.50 and 54.90 on MultiWOZ2.1 and MultiWOz2.2, respectively. |
Dynamic Oracle for Neural Machine Translation in Decoding Phase (L18-1)
Copied to clipboard
| Challenge: | Existing methods to improve NMT performance but there is a discrepancy between training and inference when decoding. |
| Approach: | They propose to use Scheduled Sampling to reduce the discrepancy between training and inference in NMT when decoding to mitigate the discrépancy. |
| Outcome: | The proposed methods improve translation quality over standard NMT system. |
ReGLA: Refining Gated Linear Attention (2025.naacl-long)
Copied to clipboard
| Challenge: | Recent advances in Large Language Models (LLMs) are known for their computational and storage requirements due to the quadratic computation complexity of softmax attention. |
| Approach: | They propose to reduce the quadratic computation complexity of softmax attention by using feature maps, normalization and the gating mechanism to improve performance. |
| Outcome: | The proposed model outperforms existing gated linear attention models in extensive tasks including training from scratch and post-linearization with continual pre-training. |
Chain of Thought with Explicit Evidence Reasoning for Few-shot Relation Extraction (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to few-shot relation extraction require training. |
| Approach: | They propose a method for few-shot relation extraction using large language models, called CoT-ER, chain-of-thought with explicit evidence reasoning. |
| Outcome: | The proposed approach achieves competitive performance compared to the fully-supervised state-of-the-art approach on the FewRel1.0 and FewRela2.0 datasets. |
Training Dynamics for Text Summarization Models (2022.findings-acl)
Copied to clipboard
| Challenge: | Pre-trained language models have shown impressive results when fine-tuned on large summarization datasets. |
| Approach: | They analyze the training dynamics for generation models, focusing on summarization . they find that a propensity to copy the input is learned early in the training process . |
| Outcome: | The proposed model learns at different stages of fine-tuning, the authors show . they show that factual errors are learnt in later stages, but not at high-loss tokens . |
Debiased Orthogonal Boundary-Driven Efficient Noise Mitigation (2026.acl-long)
Copied to clipboard
| Challenge: | Existing noise mitigation methods face limitations due to their task-specific design, model dependency, and significant computational overhead. |
| Approach: | They propose a model-agnostic noise label mitigation paradigm that uses an estimator model and a scoring function to assess the noise level of input pairs. |
| Outcome: | The proposed method is superior to existing methods in training tasks and tasks. |
EarlyBERT: Efficient BERT Training via Early-bird Lottery Tickets (2021.acl-long)
Copied to clipboard
| Challenge: | Large-scale pre-trained language models require enormous computational resources and long training time. |
| Approach: | They propose an algorithm to reduce inference time and train large NLP models by slimming the self-attention and fully-connected sub-layers inside a transformer. |
| Outcome: | The proposed algorithm achieves comparable performance to standard BERT with 35 45% less training time. |
Improving Non-autoregressive Neural Machine Translation with Monolingual Data (2020.acl-main)
Copied to clipboard
| Challenge: | Neural machine translation is usually done via knowledge distillation from an autoregressive (AR) model. |
| Approach: | They leverage large monolingual corpora to improve the NAR model's performance by transferring the autoregressive model' s generalization ability while preventing overfitting. |
| Outcome: | The proposed methods on the WMT14 En-De and WMT16 En-Ro news translation tasks show that monolingual data augmentation improves the NAR model to approach the teacher AR model’s performance. |
Complexity-Guided Curriculum Learning for Text Graphs (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Curriculum learning is a systematic approach to training that refines training progressively and tailors training to task requirements. |
| Approach: | They propose a curriculum learning approach that employs "spaced repetition" and complexity formalisms to guide the training process. |
| Outcome: | The proposed model gains more and uses less data, and the best curricula are equally effective. |
Data Rejuvenation: Exploiting Inactive Training Examples for Neural Machine Translation (2020.emnlp-main)
Copied to clipboard
| Challenge: | Large-scale training datasets make training neural machine translation models difficult. |
| Approach: | They propose to identify inactive training examples which contribute less to the model performance and introduce data rejuvenation to improve NMT models' training. |
| Outcome: | The proposed framework stabilizes and accelerates the training process of NMT models, resulting in models with better generalization capability. |
Measuring and Mitigating Local Instability in Deep Neural Networks (2023.findings-acl)
Copied to clipboard
| Challenge: | Uncertain details like random initialization can change the outputs of a trained system with potentially disastrous consequences. |
| Approach: | They propose a model stability problem by studying how the predictions of a deep neural network change as a consequence of stochasticity in the training process. |
| Outcome: | The proposed method outperforms data-agnostic methods and is 90% cheaper than the gold standard. |
AdaptThink: Reasoning Models Can Learn When to Think (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in large reasoning models have demonstrated remarkable capabilities in tackling complex tasks. |
| Approach: | They propose an algorithm to teach reasoning models to choose the optimal thinking mode based on problem difficulty. |
| Outcome: | The proposed algorithm reduces the average response length and improves accuracy on three math datasets. |
Training Mixed-Domain Translation Models via Federated Learning (2022.naacl-main)
Copied to clipboard
| Challenge: | Experimental results show that neural machine translation engines built via FL can be easily adapted when an FL-based aggregation is applied to fuse different domains. |
| Approach: | They propose to use federated learning to fuse mixed-domain translation models with a centralized aggregation to improve their performance. |
| Outcome: | The proposed model can be easily adapted to a mixed-domain translation model with slight modifications in the training process and perform on par with state-of-the-art training models. |
Devil’s Advocate: Novel Boosting Ensemble Method from Psychological Findings for Text Classification (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing ensemble methods that combine submodels to create a composite model can improve model performance by diminishing model bias and variance. |
| Approach: | They propose a method which uses a deliberately dissenting model to force other submodels within the ensemble to better collaborate. |
| Outcome: | The proposed method shows comparable or improved performance on 5 text classification tasks when compared to conventional methods. |
Non-Parametric Adaptation for Neural Machine Translation (N19-1)
Copied to clipboard
| Challenge: | Neural Networks trained with gradient descent are susceptible to catastrophic forgetting due to parameter shift during the training process. |
| Approach: | They propose a semi-parametric approach that relies on local phrase level similarities to retrieve neighboring phrases that are useful for translation even when overall sentence similarity is low. |
| Outcome: | The proposed approach performs well on a heterogeneous dataset with WMT, IWSLT, JRC-Acquis and OpenSubtitles. |
Online Distilling from Checkpoints for Neural Machine Translation (N19-1)
Copied to clipboard
| Challenge: | Existing neural machine translation models have a deep structure with large amounts of parameters, making them hard to train. |
| Approach: | They propose an online method to generate a teacher model from checkpoints . they show steady improvement over a strong self-attention-based baseline system . |
| Outcome: | The proposed method improves on-the-fly on several datasets and language pairs. |
Single Model Ensemble for Subword Regularized Models in Low-Resource Machine Translation (2022.findings-acl)
Copied to clipboard
| Challenge: | Existing subword regularizations use multiple segmentations during training but only use one segmentation in inference. |
| Approach: | They propose an inference strategy that uses multiple subword segmentations to solve this discrepancy in the training process and inference. |
| Outcome: | The proposed strategy reduces the cost of training and improves the performance of models trained with subword regularization in low-resource machine translation tasks. |
Referring to what you know and do not know: Making Referring Expression Generation Models Generalize To Unseen Entities (2020.coling-main)
Copied to clipboard
| Challenge: | Data-to-text Natural Language Generation (NLG) is a computational process of generating natural language from non-linguistic data. |
| Approach: | They propose two extensions to a state-of-the-art encoder-decoder REG model that generates referring expressions to unseen entities. |
| Outcome: | The proposed model generates more meaningful referring expressions to unseen entities than the original system and related work. |
A Graph Enhanced BERT Model for Event Prediction (2022.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to predict subsequent events use sparsity of event graph to improve performance. |
| Approach: | They propose to automatically build event graph using a BERT model by adding a structured variable to the model to learn to predict event connections. |
| Outcome: | The proposed model outperforms state-of-the-art models on two event prediction tasks. |
Give the Truth: Incorporate Semantic Slot into Abstractive Dialogue Summarization (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Abstractive dialogue summarization suffers from a lot of factual errors due to scattered salient elements in multi-speaker information interaction process. |
| Approach: | They propose a slot-driven beam search algorithm to give priority to generating salient elements in a limited length by "filling-in-the-blanks". |
| Outcome: | The proposed algorithm improves the slot-driven beam search algorithm on different types of factual errors and human evaluation further verifies the results. |
An Embarrassingly Simple Approach for Transfer Learning from Pretrained Language Models (N19-1)
Copied to clipboard
| Challenge: | Existing transfer learning methods employ language models pretrained on large generic corpora, but results come at a high computational cost and require task-specific architectures. |
| Approach: | They propose a transfer learning approach that combine a task-specific optimization function with an auxiliary language model objective, which is adjusted during the training process. |
| Outcome: | The proposed method surpasses well established transfer learning methods with greater level of complexity on a variety of affective and text classification tasks surpassing well established methods with higher level of difficulty. |
Progressive Class Semantic Matching for Semi-supervised Text Classification (2022.naacl-main)
Copied to clipboard
| Challenge: | Recent semi-supervised learning methods have achieved impressive performance . semi-controlled learning can be used to reduce the annotation cost of text classifiers . |
| Approach: | They propose a semi-supervised learning process that builds a standard K-way classifier and a matching network for the input text and the Class Semantic Representation (CSR). |
| Outcome: | The proposed method improves baselines and overall is more stable. |
Intent Contrastive Learning Based on Multi-view Augmentation for Sequential Recommendation (2025.coling-main)
Copied to clipboard
Bo Pei, Yingzheng Zhu, Guangjin Wang, Huajuan Duan, Wenya Wu, Fuyong Xu, Yizhao Zhu, Peiyu Liu, Ran Lu
| Challenge: | Existing work on intent-related models fails to capture long-term dependencies in user behavior and fails to effectively utilize item relevance. |
| Approach: | They propose a sequential recommendation framework that combine temporal variability with position encoding that has extrapolation properties to encode sequences, thereby expanding the model’s view of user behavior. |
| Outcome: | The proposed model improves on three real datasets by 0.8% to 14.7% compared to baselines. |
PLATO-2: Towards Building an Open-Domain Chatbot via Curriculum Learning (2021.findings-acl)
Copied to clipboard
| Challenge: | PLATO-2 is a high-quality open-domain chatbot that can generate one-to-many mappings and improve response quality. |
| Approach: | They propose a curriculum learning process to build a high-quality open-domain chatbot . they use a coarse-grained generation model and latent variables to train a generative model . |
| Outcome: | The proposed model improves on Chinese and English data and can generate diverse responses and select the best response. |
H-FND: Hierarchical False-Negative Denoising for Distant Supervision Relation Extraction (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing work on distant supervision denoising introduces false-positive (FP) and falsenegative (FN) training instances to the generated datasets. |
| Approach: | They propose a hierarchical false-negative denoising framework for distant supervision relation extraction that denoises false-positive and false- negative training instances. |
| Outcome: | The proposed framework can revise FN instances correctly and maintain high F1 scores even when 50% of instances have been turned into negatives. |
Entity-level Factual Consistency of Abstractive Text Summarization (2021.eacl-main)
Copied to clipboard
Feng Nan, Ramesh Nallapati, Zhiguo Wang, Cicero Nogueira dos Santos, Henghui Zhu, Dejiao Zhang, Kathleen McKeown, Bing Xiang
| Challenge: | Existing models exhibit entity hallucination, generating names of entities that are not present in the source document. |
| Approach: | They propose to use entity-level factual consistency to improve model quality . they propose to filter the training data to reduce entity hallucination problem . |
| Outcome: | The proposed model can reduce the entity hallucination problem by filtering the training data. |
Wasserstein Distance Regularized Sequence Representation for Text Matching in Asymmetrical Domains (2020.emnlp-main)
Copied to clipboard
| Challenge: | Asymmetrical text matching is a fundamental problem in information retrieval and natural language processing. |
| Approach: | They propose a method that regularizes features vectors projected from different domains . WD-Match can be used to improve different text matching methods . |
| Outcome: | The proposed method outperforms existing methods and benchmarks on four datasets. |
Posterior Calibrated Training on Sentence Classification Tasks (2020.acl-main)
Copied to clipboard
| Challenge: | Existing methods for posterior calibration have been used to correct poorly calibrated posterior probabilities. |
| Approach: | They propose a posterior calibration procedure that optimizes posterior probability distributions while minimizing calibration errors. |
| Outcome: | The proposed procedure reduces calibration error and improves performance on both objectives. |
SLoRA: Balancing Plasticity and Forgetting in Large Language Models for Continual Learning (2026.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) have achieved remarkable success across diverse tasks through large-scale pretraining. |
| Approach: | They propose a framework that filters noisy components from LoRA updates via subspace similarity with the base model. |
| Outcome: | The proposed framework improves accuracy by 12%, reduces forgetting by 29%, and filters out over 30% of LoRA parameters identified as noisy. |
Simultaneous Translation Policies: From Fixed to Adaptive (2020.acl-main)
Copied to clipboard
| Challenge: | Adaptive policies can balance translation quality and latency based on context information . previous methods on obtaining adaptive policies rely on complicated training process . |
| Approach: | They propose to obtain adaptive policies by a simple heuristic composition of fixed policies . they propose to use a heurism to obtain policies that can outperform fixed ones . |
| Outcome: | Experiments on Chinese -> English and German -> english show that adaptive policies outperform fixed policies by up to 4 BLEU points for the same latency. |
QuZO: Quantized Zeroth-Order Fine-Tuning for Large Language Models (2025.emnlp-main)
Copied to clipboard
Jiajun Zhou, Yifan Yang, Kai Zhen, Ziyue Liu, Yequan Zhao, Ershad Banijamali, Athanasios Mouchtaris, Ngai Wong, Zheng Zhang
| Challenge: | Large Language Models (LLMs) are quantized to lower precision to reduce memory cost and latency in inference. |
| Approach: | They propose a quantized zeroth-order framework for fine-tuning Large Language Models (LLMs) using low-precision forward passes. |
| Outcome: | The proposed method achieves comparable results to first-order methods in FP8 and superior accuracy in INT8 and INT4 training. |
Standardizing Distress Analysis: Emotion-Driven Distress Identification and Cause Extraction (DICE) in Multimodal Online Posts (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for identifying hate speech have been limited to analyzing textual content. |
| Approach: | They propose a method for distress identification and cause extraction from social media posts using emotional information. |
| Outcome: | The proposed method improves F1 and ROS scores by 1.95% and 3% relative to the best-performing baseline. |
On Long-Tailed Phenomena in Neural Machine Translation (2020.findings-emnlp)
Copied to clipboard
| Challenge: | State-of-the-art Neural Machine Translation models struggle with generating low-frequency tokens, tackling which remains a major challenge. |
| Approach: | They propose a loss function to better adapt model training to structural dependencies of conditional text generation by incorporating inductive biases of beam search into the training process. |
| Outcome: | The proposed method leads to significant gains over cross-entropy across different language pairs, especially on the generation of low-frequency words. |
Transferring Knowledge from Structure-aware Self-attention Language Model to Sequence-to-Sequence Semantic Parsing (2022.coling-1)
Copied to clipboard
| Challenge: | Semantic parsing aims to map a natural language sentence into a machine executable formal representation. |
| Approach: | They propose a structure-aware self-attention language model to capture structural information of target representations and propose incorporating it into a seq2seq model. |
| Outcome: | The proposed model improves the baseline model on four semantic parsing and Python code generation tasks. |
Data Annealing for Informal Language Understanding Tasks (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing models that improve formal and informal language understanding tasks do not transfer to informal data directly. |
| Approach: | They propose a data annealing transfer learning procedure to bridge the performance gap on informal natural language understanding tasks. |
| Outcome: | The proposed procedure outperforms state-of-the-art models on three common tasks. |
Inductive Relation Prediction with Logical Reasoning Using Contrastive Representations (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for relation prediction in knowledge graphs (KGs) are limited by the inductive setting because entities in training process are finite. |
| Approach: | They propose a graph convolutional network-based model LogCo with logical reasoning by contrastive representations that extracts subgraphs and relational paths between two entities to supply the entity-independence. |
| Outcome: | The proposed model outperforms existing methods on twelve inductive datasets. |
A Span-level Bidirectional Network for Aspect Sentiment Triplet Extraction (2022.emnlp-main)
Copied to clipboard
| Challenge: | Aspect Sentiment Triplet Extraction (ASTE) is a new fine-grained sentiment analysis task . recent studies have focused on solving aspects term extraction, opinion term extraction and aspect-level sentiment classification tasks individually or in combination of two subtasks. |
| Approach: | They propose a span-level bidirectional network which utilizes all possible spans as input and extracts triplets from spans bidirectionally. |
| Outcome: | The proposed framework outperforms state-of-the-art methods and improves performance . it can extract triplets of aspect terms, sentiments, and opinion terms from review sentences . |
CLHA: A Simple Yet Effective Contrastive Learning Framework for Human Alignment (2024.lrec-main)
Copied to clipboard
Feiteng Fang, Liang Zhu, Xi Feng, Jinchang Hou, Qixuan Zhao, Chengming Li, Xiping Hu, Ruifeng Xu, Min Yang
| Challenge: | Large language models (LLMs) have attracted considerable attention from academic and industrial communities due to their outstanding performance in various natural language processing tasks. |
| Approach: | They propose a Contrastive Learning Framework for Human Alignment to evaluate the noise within the data and dynamically adjust the training process. |
| Outcome: | The proposed framework surpasses other algorithms in terms of reward model scores, automatic evaluations, and human assessments on the widely used dataset "Helpful and Harmless" |
NCLS: Neural Cross-Lingual Summarization (D19-1)
Copied to clipboard
| Challenge: | Existing approaches to cross-lingual summarization divide the task into two steps: summarizing and translation. |
| Approach: | They propose to integrate two related tasks into the training process of CLS under multi-task learning to improve cross-lingual summarization. |
| Outcome: | The proposed framework improves on English-to-Chinese and Chinese-to English CLS human-corrected test sets. |
TAG: Gradient Attack on Transformer-based Language Models (2021.findings-emnlp)
Copied to clipboard
Jieren Deng, Yijue Wang, Ji Li, Chenghong Wang, Chao Shang, Hang Liu, Sanguthevar Rajasekaran, Caiwen Ding
| Challenge: | Recent studies show that publicly shared gradients in the training process can reveal the private training data to a third-party. |
| Approach: | They propose a gradient attack algorithm to reconstruct the local training data using GLUE benchmarks. |
| Outcome: | The proposed algorithm achieves 1.5x recover rate and 2.5x ROUGE-2 over previous methods without the need of ground truth label. |
Zero-Shot Cross-Domain Dialogue State Tracking via Dual Low-Rank Adaptation (2024.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to zero-shot dialogue state tracking (DST) involve embedding prompts into language models, but these methods have inherent limitations. |
| Approach: | They propose a plug-and-play architecture designed for zero-shot dialogue state tracking (DST) dual low-rank adaptation targets dialogue context processing and prompt optimization without incurring additional inference latency. |
| Outcome: | The proposed architecture outperforms baseline methods on multi-domain datasets and the MultiWOZ dataset. |
Fusion or Defusion? Flexible Vision-and-Language Pre-Training (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to vision-and-language pretraining (VLP) lack effectiveness and efficiency in downstream multimodal tasks. |
| Approach: | They propose a flexible vision-and-language pre-training model by incorporating cross-modal fusions into a dual-encoder architecture and a cross-module knowledge transfer strategy to guide the training process. |
| Outcome: | The proposed model is well-equipped with effectiveness and efficiency compared with other strong VLP models. |
Leveraging Declarative Knowledge in Text and First-Order Logic for Fine-Grained Propaganda Detection (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for fine-grained propaganda detection are not based on input-output data, but instead use declarative knowledge to detect propagandistic text fragments. |
| Approach: | They propose a method to inject declarative knowledge of fine-grained propaganda techniques into training data to get better representations of propagandistic texts. |
| Outcome: | The proposed method achieves superior performance on a large dataset for propaganda detection. |
Revisiting Robust Neural Machine Translation: A Transformer Case Study (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Recent advances in NMT have shown promising results but are vulnerable to noise. |
| Approach: | They propose a data-driven technique called Target Augmented Fine-tuning to incorporate noise during training. |
| Outcome: | The proposed techniques perform with no degradation where up to 10% of entire test words are infected by noise. |
Breaking the Stage Barrier: A Novel Single-Stage Approach to Long Context Extension for Large Language Models (2025.coling-main)
Copied to clipboard
Haoran Lian, Junmin Chen, Wei Huang, Yizhe Xiong, Wenping Hu, Guiguang Ding, Hui Chen, Jianwei Niu, Zijia Lin, Fuzheng Zhang, Di Zhang
| Challenge: | Recent studies show that Large language models struggle with handling long token sequences due to limited training context size. |
| Approach: | They propose a single-stage continual pretraining method to equip LLMs with long context modeling capabilities. |
| Outcome: | The proposed method outperforms existing methods on 4 language modeling benchmarks. |
Natural Language Video Localization with Learnable Moment Proposals (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for video moment localization have poor performance due to predefined rules. |
| Approach: | They propose a model with a fixed set of learnable moment proposals with 'border-aware loss' they propose to localize the video moment corresponding to the query by locating the start and end timestamps in an untrimmed video. |
| Outcome: | The proposed model outperforms state-of-the-art models on two challenging benchmarks. |
VersaTune: An Efficient Data Composition Framework for Training Multi-Capability LLMs (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing work focuses on domain-specific enhancements during fine-tuning, the challenge of which lies in catastrophic forgetting of knowledge across other domains. |
| Approach: | They propose a data composition framework that allows LLMs to enhance their multi-domain capabilities during supervised fine-tuning. |
| Outcome: | The proposed framework improves multi-domain fostering performance by 29.77% compared to uniform weights. |
AGGC: Adaptive Group Gradient Clipping for Stabilizing Large Language Model Training (2026.findings-acl)
Copied to clipboard
| Challenge: | Adaptive group-wise gradient clipping (AGGC) is a new approach to stabilize training of Large Language Models. |
| Approach: | They propose a method to stabilize gradient clipping by partitioning parameters into groups based on functional types and a time-dependent scheduling mechanism to balance exploration and convergence. |
| Outcome: | The proposed algorithm outperforms standard LoRA and achieves 72.93% accuracy . it can be integrated into existing pipelines with negligible overhead. |
LoSiA: Efficient High-Rank Fine-Tuning via Subnet Localization and Optimization (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods perform extensive matrix multiplications in domain specialization tasks, resulting in computational inefficiency and sub-optimal fine-tuning performance. |
| Approach: | They propose a method that localizes and optimizes critical parameters during training . they propose 'LoSiA-Pro' which reduces training latency by 27% . |
| Outcome: | The proposed method achieves minimal performance drop compared to full fine-tuning while requiring the least training time across domain specialization and common-sense reasoning tasks. |
Generic resources are what you need: Style transfer tasks without task-specific parallel training data (2021.emnlp-main)
Copied to clipboard
| Challenge: | Text style transfer is a task aimed at converting a text of one style into another while preserving its content. |
| Approach: | They propose a multi-step procedure which builds on a generic pre-trained sequence-to-sequence model and an iterative back-translation approach to train two models in a transfer direction. |
| Outcome: | The proposed method outperforms existing unsupervised approaches on the two most popular style transfer tasks: formality transfer and polarity swap. |
Wav2Prompt: End-to-End Speech Prompt Learning and Task-based Fine-tuning for Text-based LLMs (2025.naacl-long)
Copied to clipboard
| Challenge: | Text-based large language models (LLMs) can be applied to a wide range of tasks without being explicitly trained. |
| Approach: | They propose a method which integrates spoken input with a text-based large language model (LLM) it takes LLM token embeddings as training targets and utilises a continuous integrate-and-fire mechanism for explicit speech-text alignment. |
| Outcome: | The proposed model can be applied to speech translation, speech understanding and spoken-query-based question answering tasks. |
Mitigating Data Poisoning in Text Classification with Differential Privacy (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Data poisoning attacks can plant a backdoor in a model by injecting poisoned examples into training data, causing the model to misclassify test instances which include a specific pattern. |
| Approach: | They propose a generic defence mechanism that makes training robust to poisoning attacks by smoothing the gradient from each training example. |
| Outcome: | The proposed method is highly effective in mitigating, or even eliminating, poisoning attacks on text classification, with only a small cost in predictive accuracy. |
𝒮2IT: Stepwise Syntax Integration Tuning for Large Language Models in Aspect Sentiment Quad Prediction (2025.findings-naacl)
Copied to clipboard
| Challenge: | Aspect Sentiment Quad Prediction (ASQP) is an extractive task that focuses on predicting tuples of sentiment-related elements from a given text. |
| Approach: | They propose a stepwise syntax integration tuning framework that integrates syntactic structure knowledge into LLMs through a multi-step tuning process. |
| Outcome: | The proposed framework integrates syntactic structure knowledge into large language models . it decomposes the quadruple generation task into two stages . the proposed framework significantly improves state-of-the-art performance across multiple datasets . |
Padding Tone: A Mechanistic Analysis of Padding Tokens in T2I Models (2025.naacl-long)
Copied to clipboard
| Challenge: | Text-to-image (T2I) diffusion models rely on encoded prompts to guide the image generation process. |
| Approach: | They conduct the first in-depth analysis of the role padding tokens play in T2I diffusion models by using two causal techniques to analyze how information is encoded in the representation of tokens across different components of the pipeline. |
| Outcome: | The proposed techniques reveal that padding tokens may affect the model’s output during text encoding, during the diffusion process, or be effectively ignored. |
CaLcs: Continuously Approximating Longest Common Subsequence for Sequence Level Optimization (D18-1)
Copied to clipboard
| Challenge: | Maximum-likelihood estimation (MLE) is widely used for text-generation based natural language processing applications. |
| Approach: | They propose a method to train models with maximum-likelihood estimation using a differentiable surrogate of longest common subsequence measure that captures sequence-level structure similarity. |
| Outcome: | Experimental results show that the proposed approach improves on the current MLE approach for downstream tasks like text summarization and machine translation. |
Data Cartography for Low-Resource Neural Machine Translation (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to improve machine translation (MT) in low-resource settings are limited in the number of languages spoken in the world. |
| Approach: | They apply cartography techniques to characterize the contribution of training samples in two low-resource MT tasks (Swahili-English and Turkish-English) they argue that data augmentation strategies for low-Resource ML would benefit from model-in-the-loop strategies to maximize improvements. |
| Outcome: | The proposed methods show that training samples contribute to model training in low-resource MT tasks, albeit not uniformly throughout the training process. |
Efficient Continue Training of Temporal Language Model with Structural Information (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing temporal language models are limited by the superficial temporal information brought by timestamps, which fails to learn the inherent changes of linguistic components. |
| Approach: | They propose a method that captures syntactically changed tokens and captures the relationship between the time prefix and tokens. |
| Outcome: | The proposed method outperforms existing temporal language models on two datasets and three tasks. |
Contextual Knowledge Learning for Dialogue Generation (2023.acl-long)
Copied to clipboard
| Challenge: | Incorporating conversational context and knowledge into dialogue generation models has been essential for improving the quality of the generated responses. |
| Approach: | They propose a method to incorporate conversational context and knowledge into dialogue generation models . they use Latent Vectors to capture the relationship between context and knowing . |
| Outcome: | The proposed approach improves performance with two standard datasets and human evaluations. |
OpticE: A Coherence Theory-Based Model for Link Prediction (2022.coling-1)
Copied to clipboard
| Challenge: | Knowledge representation learning is a key step required for link prediction tasks with knowledge graphs (KGs). |
| Approach: | They propose a new embedding approach based on the physical phenomenon of optical interference to reduce the semantic ambiguity in KGs. |
| Outcome: | The proposed model can compete with existing methods on KG benchmarks. |
ARAML: A Stable Adversarial Training Framework for Text Generation (D19-1)
Copied to clipboard
| Challenge: | Existing generative adversarial networks suffer from the instability of reinforcement learning training algorithms such as policy gradient, leading to unstable performance. |
| Approach: | They propose a framework where the discriminator assigns rewards to samples acquired from a stationary distribution near the data rather than the generator’s distribution. |
| Outcome: | The proposed framework outperforms state-of-the-art text GANs with a more stable training process. |
Modeling Document-Level Context for Event Detection via Important Context Selection (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for Event Detection (ED) do not encode long-range document-level context . e.g., BERT cannot encode long text-level contextual information . |
| Approach: | They propose a method to model document-level context for Event Detection using transformer-based language models. |
| Outcome: | The proposed model can predict event prediction of target sentence in document-level context . the proposed model is effective on multiple benchmark datasets . |
Revealing the Parametric Knowledge of Language Models: A Unified Framework for Attribution Methods (2024.acl-long)
Copied to clipboard
| Challenge: | Language Models acquire parametric knowledge from their training process, embedding it within their weights. |
| Approach: | They propose a new evaluation framework to quantify and compare the knowledge revealed by Instance Attribution and Neuron Attributions. |
| Outcome: | The proposed evaluation framework compares the knowledge revealed by IA and NA with that of neuron attribution methods. |
Light-PEFT: Lightening Parameter-Efficient Fine-Tuning via Early Pruning (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for fine-tuning large language models are inefficient and redundant . a light-PEFT framework can be used to prune redundant parameters during training . |
| Approach: | They propose a parameter-efficient fine-tuning framework that freezes most parameters of the foundation model and finetuns only a small number of parameters. |
| Outcome: | The proposed framework achieves training and inference speedup, reduces memory usage, and maintains comparable performance and plug-and-play feature of PEFT. |
Distilling Causal Effect of Data in Continual Few-shot Relation Learning (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing methods for learning relational patterns from data are prone to catastrophic forgetting issues due to limited number of samples and continual training mode. |
| Approach: | They propose a unified causal framework for CFRL to restore causal effects from old data . they establish two additional causal paths from old to predictions by colliding with old data separately in the old feature space. |
| Outcome: | The proposed method is superior to existing state-of-the-art methods in CFRL task settings. |
Deep Weighted MaxSAT for Aspect-based Opinion Extraction (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing deep learning models lack the capability to encode explicit domain knowledge to model complex causal relationships among variables. |
| Approach: | They propose a model that uses a weighted version of MaxSAT to model logic inference . they propose to use this model to rectify erroneous predictions from deep neural networks . |
| Outcome: | The proposed model combines the benefits of high-level feature learning, knowledge reasoning, and structured learning with observable performance gain for aspect-based opinion extraction. |
CoBa: Convergence Balancer for Multitask Finetuning of Large Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing multi-task learning approaches for large language models fall short due to computational intensive or lack of simultaneous task convergence. |
| Approach: | They propose a new multi-task learning approach that dynamically adjusts task weights during the training process, ensuring that the validation loss of all tasks progresses towards convergence at an even pace. |
| Outcome: | The proposed approach improves the performance of large language models by up to 13% compared to the second-best approaches. |
TF-LM: TensorFlow-based Language Modeling Toolkit (L18-1)
Copied to clipboard
| Challenge: | Existing deep learning tools offer building blocks but training and building models takes time and knowledge. |
| Approach: | They propose to make available LSTM language models trained on Dutch texts and English benchmarks. |
| Outcome: | The proposed model can be used to test the perplexity, predict the next word(s), re-score hypotheses or generate debugging files for interpolation with n-gram models. |
Capturing Relations between Scientific Papers: An Abstractive Model for Related Work Section Generation (2021.acl-long)
Copied to clipboard
| Challenge: | Existing related work generation models are inflexible and extract sentences from multiple papers to form a related work discussion. |
| Approach: | They propose a Relation-aware Related work generator which generates an abstractive related work from the given multiple scientific papers in the same research area. |
| Outcome: | The proposed model improves over existing models and can be used to familiarize researchers with the state of the art in the field. |
R-AT: Regularized Adversarial Training for Natural Language Understanding (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Currently, adversarial training is a popular and powerful regularization method in the natural language domain. |
| Approach: | They propose to regularize adversarial training via dropout by perturbing word embeddings . they find that R-AT can improve many models by reducing adversariality . |
| Outcome: | The proposed method can reduce the inconsistency between training and testing of models with dropout. |
Parsing Natural Language into Propositional and First-Order Logic with Dual Reinforcement Learning (2022.coling-1)
Copied to clipboard
Xuantao Lu, Jingping Liu, Zhouhong Gu, Hanwen Tong, Chenhao Xie, Junyang Huang, Yanghua Xiao, Wenguang Wang
| Challenge: | Existing methods to parse natural language into structured logical expressions have limitations due to paucity of labeled data. |
| Approach: | They propose a scoring model to automatically learn a model-based reward . they also propose introducing a Chinese-PL/FOL dataset to compensate for paucity of labeled data . |
| Outcome: | The proposed model outperforms competitors on several datasets. |
BLIND: Bias Removal With No Demographics (2023.acl-long)
Copied to clipboard
| Challenge: | Numerous methods to mitigate social biases require prior knowledge of the demographics in the dataset, such as gender or race. |
| Approach: | They propose a method for bias removal without prior knowledge of demographics in the dataset. |
| Outcome: | Experiments with racial and gender biases in sentiment classification and occupation classification tasks show that BLIND mitigates biase . BLINT is competitive with methods that require demographic information and sometimes surpasses them. |
SALMON: A Structure-Aware Language Model with logicality and densification strategy for Temporal Knowledge Graph Reasoning (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Temporal knowledge graph reasoning (TKGR) is a crucial task that involves reasoning at known timestamps to complete the future facts. |
| Approach: | They propose a temporal knowledge graph reasoning model with logicality and densification strategy that captures temporal evolving pattern and structural information in TKGs. |
| Outcome: | The proposed model outperforms the state-of-the-art models and is based on a structure-aware language model with logicality and densification strategy. |
Logic Unveils Truth, While Disguise Obscures It: Transition Logic Augmented Response Selection for Multi-Turn Dialogue (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods of negative samples tend to yield false negatives due to one-to-many property in open-domain dialogue. |
| Approach: | They propose a sequential variational ladder auto-encoder to capture one-to-many transition pattern of multiple characteristics in open-domain dialogue. |
| Outcome: | The proposed approach improves the performance of a retrieval dialogue system on two benchmarks. |
Enhancing Parameter-efficient Fine-tuning with Simple Calibration Based on Stable Rank (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing methods for lightweight fine-tuning are ineffective in low-resource settings but fail in high-resourced settings, leading to unreliable outcomes. |
| Approach: | They propose a calibration strategy that takes into account the inherent variance of generalization ability in model components and potential changes during the fine-tuning process. |
| Outcome: | The proposed calibration improves GLUE score by 3.1 points over the previous calibration method. |
StablePT : Towards Stable Prompting for Few-shot Learning via Input Separation (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies on prompt tuning have shown that language models can be effective few-shot learners with prompting. |
| Approach: | They propose to treat the hard prompt and soft prompt as separate inputs to mitigate noise brought by prompt initialization. |
| Outcome: | Experimental results show that the proposed method outperforms state-of-the-art methods by 6.97% in accuracy and reduces the standard deviation by 1.92 on average. |
PSP: Pre-trained Soft Prompts for Few-Shot Abstractive Summarization (2022.coling-1)
Copied to clipboard
| Challenge: | Experimental results show that our method outperforms full-model tuning in few-shot abstractive summarization tasks. |
| Approach: | They propose a soft prompts architecture with prompt pre-training and prompt fine-tuning paradigm to support few-shot abstractive summarization. |
| Outcome: | The proposed model outperforms Prompt Tuning and Profix-Tuning on CNN/DailyMail and XSum datasets and outperfies Profix Tuning by a large margin. |
Human-in-the-loop Abstractive Dialogue Summarization (2023.findings-acl)
Copied to clipboard
| Challenge: | Abstractive dialogue summarization systems are trained to maximize the likelihood of human-written summaries, but there is still a huge gap in generating high-quality summary as determined by humans. |
| Approach: | They propose to incorporate different levels of human feedback into the training process . they ask humans to highlight salient information to be included in summaries . |
| Outcome: | The proposed model captures human-written summaries and compares them with state-of-the-art models on multiple datasets. |
Sparsifying Transformer Models with Trainable Representation Pooling (2022.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to sparsify attention in the Transformer model are based on quadratic memory complexity and a lack of information for each word. |
| Approach: | They propose a method to sparsify attention in a Transformer model by learning to select the most-informative token representations during the training process. |
| Outcome: | The proposed model performs better than the current SOTA model while being 1.8 faster during training, 4.5 faster inference and 13 more efficient in the decoder. |
Unsupervised Dual Paraphrasing for Two-stage Semantic Parsing (2020.acl-main)
Copied to clipboard
| Challenge: | Existing semantic parsing frameworks rely on nontrivial human labor to generate canonical utterances. |
| Approach: | They propose a framework that uses an unsupervised paraphrase model to parse canonical utterances. |
| Outcome: | The proposed framework is effective and compatible with supervised training. |
Improving Clustering with Positive Pairs Generated from LLM-Driven Labels (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing unsupervised clustering methods lack label knowledge, resulting in suboptimal performance. |
| Approach: | They propose to use LLM-driven labels to generate positive pairs from embedded data and an embedder to obviate the need for negative pairs. |
| Outcome: | The proposed framework surpasses state-of-the-art benchmarks on a range of datasets and generates interpretable labels for improved understanding of clustering results. |
Factorized Learning Assisted with Large Language Model for Gloss-free Sign Language Translation (2024.lrec-main)
Copied to clipboard
| Challenge: | Previous Sign Language Translation methods have relied on gloss annotations to improve performance, but labeling high-quality glosses is labor-intensive and inefficient. |
| Approach: | They propose to integrate Large Language Model (LLM) into SLT by factorizing learning into two stages to improve the learning curve. |
| Outcome: | The proposed approach improves on three SLT datasets conducted under the gloss-free setting. |
Dynamic Low-rank Estimation for Transformer-based Language Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | RankDyna is a matrix decomposition method that can be used to compress Transformer-based language models. |
| Approach: | They propose a matrix decomposition method that enables dynamic rank resource allocation . they say it can outperform current SOTA methods under various parameter budget levels . |
| Outcome: | The proposed method outperforms current SOTA methods under various budget levels . the proposed method is more efficient with higher compression rates . |
Unveiling the Role of Pretraining in Direct Speech Translation (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to train direct speech-to-text translation systems are pretraining the encoder on automatic speech recognition, thus losing efficiency in the training process. |
| Approach: | They propose to change the decoder cross-attention to integrate source information from earlier steps in training. |
| Outcome: | The proposed model can achieve comparable performance to the pretrained model while reducing training time. |
Boosting LLM Translation Skills without General Ability Loss via Rationale Distillation (2025.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have demonstrated impressive performance across numerous NLP tasks, but fine-tuning them for Machine Translation (MT) often introduces catastrophic forgetting, compromising the broad general abilities of LLMs and introducing potential security risks. |
| Approach: | They propose a method that harnesses the strong generative capabilities of Large Language Models to create rationales for training data, which are then "replayed" to prevent forgetting. |
| Outcome: | The proposed approach harnesses the strong generative capabilities of LLMs to create rationales for training data, which are then “replayed” to prevent forgetting. |
Noise-Robust Training with Dynamic Loss and Contrastive Learning for Distantly-Supervised Named Entity Recognition (2023.findings-acl)
Copied to clipboard
| Challenge: | Named entity recognition (NER) is a task in natural language processing that aims at locating entity mentions in a given sentence and assigning them to certain types. |
| Approach: | They propose to use a dynamic loss function to better adapt to the changing noise during the training process and incorporate token level contrastive learning to fully utilize the noisy data. |
| Outcome: | The proposed method outperforms existing NER models on three benchmark datasets and outperformed existing models by significant margins. |
Fighting Randomness with Randomness: Mitigating Optimisation Instability of Fine-Tuning using Delayed Ensemble and Noisy Interpolation (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Despite the success of fine-tuning, it still displays model performance instability, especially with limited data. |
| Approach: | They propose a new mitigation strategy that leverages the strengths of ensembling, noise regularisation and model interpolation while retaining computational efficiency. |
| Outcome: | The proposed mitigation strategy outperforms the best performing mitigation strategy (Ensemble) while using only a fraction of its cost. |
On the Calibration of Large Language Models and Alignment (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models are becoming more popular and are proving to be reliable . however, their reliability is often understudied due to their uncertainty and complex structure . |
| Approach: | They conduct a systematic examination of the calibration of aligned language models throughout the entire construction process including pretraining and alignment training. |
| Outcome: | The results shed light on whether popular large language models are well-calibrated and how the training process influences model calibration. |
PIP: Parse-Instructed Prefix for Syntactically Controlled Paraphrase Generation (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing fine-tuning methods for this task are costly and require updating the parameters of the entire model to adapt to the newly included syntax information. |
| Approach: | They propose a method to instruct model’s encoder prefix to capture syntax-related knowledge by direct initiation and indirect optimization. |
| Outcome: | The proposed methods are 10 times more efficient and learnable than existing methods. |
Auto-Instruct: Automatic Instruction Generation and Ranking for Black-Box Language Models (2023.findings-emnlp)
Copied to clipboard
Zhihan Zhang, Shuohang Wang, Wenhao Yu, Yichong Xu, Dan Iter, Qingkai Zeng, Yang Liu, Chenguang Zhu, Meng Jiang
| Challenge: | Large language models can perform a wide range of tasks by following natural language instructions without task-specific fine-tuning. |
| Approach: | They propose a method to automatically improve the quality of LLM instructions . they leverage the generative ability of LMS to generate diverse candidate instructions based on a scoring model trained on 575 existing NLP tasks. |
| Outcome: | The proposed method surpasses human-written and LLM-generated instructions on 118 out-of-domain tasks. |
Language Modeling, Lexical Translation, Reordering: The Training Process of NMT through the Lens of Classical SMT (2021.emnlp-main)
Copied to clipboard
| Challenge: | Statistical MT decomposes the translation task into distinct components that are learned separately. |
| Approach: | They show that neural machine translation models acquire different competences over the course of training . previous work shows how to improve some of the competences in NMT by using lexical translation probabilities, phrase memories, alignment information. |
| Outcome: | The proposed model improves translation quality and word-by-word translation, while learning complex reordering patterns. |
Improving Low-Resource Machine Translation for Formosan Languages Using Bilingual Lexical Resources (2024.findings-acl)
Copied to clipboard
| Challenge: | Using bilingual lexicons for low-resource languages can improve machine translation for low resource languages. |
| Approach: | They propose to use bilingual lexicons to improve machine translation for low-resource languages . they use parallel data and bilingual dictionaries to generate pseudo-parallel sentences . |
| Outcome: | The proposed techniques improve translation between Mandarin and Formosan languages and Spanish and Nahuatl, a language pair consisting of languages from completely different language families. |
MoKA:Parameter Efficiency Fine-Tuning via Mixture of Kronecker Product Adaption (2025.coling-main)
Copied to clipboard
| Challenge: | Low-Rank Adaptation (LoRA) is one of the most popular PEFT methods . low-rank update mechanism of LoRA somewhat limits its ability to approximate full-parameter fine-tuning during training process. |
| Approach: | They propose a parameter-efficient fine-tuning framework that combines Kronecker product with the Mixture-of-Experts method to achieve parameter efficiency and better model performance. |
| Outcome: | The proposed framework outperforms existing methods on the GLUE benchmark and instruction tuning tasks for large language models. |
Latent Concept-based Explanation of NLP Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing attempts to explain deep learning models rely on input features, such as the words . however, such explanations are often less informative due to the discrete nature of words and lack of contextual verbosity. |
| Approach: | They propose a method that generates explanations for predictions based on latent concepts . they map the representations of salient input words into the training latent space . |
| Outcome: | The proposed method generates explanations for predictions based on latent concepts . it maps representations of salient input words into training latent space . |
Investigating the Impact of Model Instability on Explanations and Uncertainty (2024.findings-acl)
Copied to clipboard
| Challenge: | Explainable AI methods are typically evaluated holistically, but small perturbations to inputs can vastly distort explanations. |
| Approach: | They artificially simulate epistemic uncertainty in text input by introducing noise at inference time and measure the effect on the output of pre-trained language models. |
| Outcome: | The proposed model can detect salient tokens when uncertain, but it is not reliable when small perturbations are exposed during training. |
OmniFlatten: An End-to-end GPT Model for Seamless Voice Conversation (2025.acl-long)
Copied to clipboard
Qinglin Zhang, Luyao Cheng, Chong Deng, Qian Chen, Wen Wang, Siqi Zheng, Jiaqing Liu, Hai Yu, Chao-Hong Tan, Zhihao Du, ShiLiang Zhang
| Challenge: | Full-duplex spoken dialogue systems allow simultaneous bidirectional communication . low latency and natural interactions in full-duplice systems remains a challenge . |
| Approach: | They propose a multi-stage post-training scheme that adapts a text large language model into a speech-text dialogue LLM. |
| Outcome: | The proposed model can model human conversation behaviors with low latency and natural interactions with low delay. |
QCRD: Quality-guided Contrastive Rationale Distillation for Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent research has focused on smaller, task-specific models enhanced by distilling knowledge from LLMs, but the diversity and quality of negative knowledge remains understudied. |
| Approach: | They propose a quality-guided contrastive rationale distillation framework that aims to enhance reasoning capabilities through contrastive knowledge learning. |
| Outcome: | The proposed method consistently outperforms existing distillation techniques yielding higher-quality rationales. |
Generating Labeled Data for Relation Extraction: A Meta Learning Approach with Joint GPT-2 Training (2023.findings-acl)
Copied to clipboard
| Challenge: | Relation Extraction (RE) is the task of identifying semantic relation between entities mentioned in text. |
| Approach: | They propose a framework to automatically generate labeled data for Relation Extraction . they propose 'reward function' to update pre-trained language model for RE . |
| Outcome: | The proposed framework generates labeled data for relation extraction using a pre-trained language model and a meta learning approach to improve the generated samples. |
Conan-Embedding-v2: Training an LLM from Scratch for Text Embeddings (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies use LoRA to fine-tune existing LLMs, but this is limited by the data and training gap between them and embedding models. |
| Approach: | They propose a new 1.4B-parameter LLM trained from scratch and fine-tuned as a text embedder that integrates embeddings across different languages. |
| Outcome: | The proposed model improves performance on the Massive Text Embedding Benchmark (MTEB) and Chinese MTEB (May 19, 2025). |
Enhancing Generative Retrieval with Reinforcement Learning from Relevance Feedback (2023.emnlp-main)
Copied to clipboard
| Challenge: | End-to-end generative retrieval models produce document identifiers in response to a query . however, this approach has two challenges: an overemphasis on top-1 results at the expense of overall ranking quality. |
| Approach: | They propose a generative retrieval model with reinforcement learning from relevance feedback to align token-level docid generation with document-level relevance estimation. |
| Outcome: | The proposed model aligns token-level docid generation with document-level relevance estimation. |
Instance Regularization for Discriminative Language Model Pre-training (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies have optimized independent strategies of ennoising or denosing . Existing methods treat training instances equally throughout the training process . |
| Approach: | They propose to use ennoising and denoising to train discriminative pre-trained language models . they propose to model the complexity of restoring the original sentences from corrupted ones . |
| Outcome: | Experimental results show that the proposed method improves pre-training efficiency, effectiveness, and robustness. |
Pierce the Mists, Greet the Sky: Decipher Knowledge Overshadowing via Knowledge Circuit Analysis (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are hampered by hallucinations, a particularly challenging variant, knowledge overshadowing, which can lead to erroneous outputs even with high-quality training data. |
| Approach: | They propose a framework to analyze and detect knowledge overshadowing by using knowledge circuit analysis to dissect the function of key components in the circuit and how attention pattern dynamics contribute to the phenomenon. |
| Outcome: | Extensive experiments show that the framework can detect and analyze knowledge overshadowing and improves on existing models. |
Improving the Robustness of Large Language Models via Consistency Alignment (2024.lrec-main)
Copied to clipboard
Yukun Zhao, Lingyong Yan, Weiwei Sun, Guoliang Xing, Shuaiqiang Wang, Chong Meng, Zhicong Cheng, Zhaochun Ren, Dawei Yin
| Challenge: | Large language models have shown tremendous success in following user instructions and generating helpful responses, but their robustness is still far from optimal. |
| Approach: | They propose a two-stage training framework that helps a model generalize on following instructions via similar instruction augmentations. |
| Outcome: | The proposed training framework improves diversity and aligns the model with human expectations by differentiating subtle differences in similar responses. |
Enhancing Training Data Attribution for Large Language Models with Fitting Error Consideration (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) are difficult to interpret due to their black-box nature and randomness. |
| Approach: | They propose a new method which enhances influence functions by addressing fitting errors by eliminating knowledge bias present in the base model before fine-tuning. |
| Outcome: | The proposed method outperforms existing methods and achieves an average AUC of 91.64%. |
FedDQC: Data Quality Control in Federated Instruction-tuning of Large Language Models (2025.findings-acl)
Copied to clipboard
| Challenge: | Federated Learning (FL) enables privacy-preserving collaborative instruction tuning of large language models. |
| Approach: | They propose a federated instruction tuning framework with dynamic data quality control to solve this problem. |
| Outcome: | The proposed framework improves performance on mixed-quality datasets on synthetic and real-world datasets. |
Prompt Tuning Pushes Farther, Contrastive Learning Pulls Closer: A Two-Stage Approach to Mitigate Social Biases (2023.acl-long)
Copied to clipboard
| Challenge: | Existing debiasing techniques use Counterfactual Data Augmentation (CDA) to balance the training corpus, but this technique slightly modifies the original corpus limiting the representation distance between different demographic groups. |
| Approach: | They propose a two-stage debiasing model using Contrastive learning with Continuous Prompt Augmentation to mitigate social biases in PLMs’ encoding. |
| Outcome: | The proposed model outperforms baselines in terms of debiasing performance while maintaining the language modeling capability of PLMs. |
Doolittle: Benchmarks and Corpora for Academic Writing Formalization (2023.emnlp-main)
Copied to clipboard
Shizhe Diao, Yongyu Lei, Liangming Pan, Tianqing Fang, Wangchunshu Zhou, Sedrick Keh, Min-Yen Kan, Tong Zhang
| Challenge: | Existing methods of language refinement focus on narrow, specific linguistic features within isolated sentences, such as grammatical errors and improper word use. |
| Approach: | They propose a task to improve the overall quality of academic writing at paragraph level by integrating automatic feedback into the training process. |
| Outcome: | The proposed task improves the overall quality of formal academic writing at the paragraph level. |
IT2ACL Learning Easy-to-Hard Instructions via 2-Phase Automated Curriculum Learning for Large Language Models (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing studies have focused on the pre-training phase of large language models, but this study focuses on the learning phase of pre-trained LLMs. |
| Approach: | They propose a 2-phase automated curriculum learning guided instruction tuning framework that learns easy-to-hard instructions in a self-adjusting dynamic manner. |
| Outcome: | The proposed framework unlocks latent ability in pre-trained large language models and achieving superior performance across diverse tasks. |
Roles of Scaling and Instruction Tuning in Language Perception: Model vs. Human Attention (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent large language models (LLMs) have shown strong abilities to understand natural language, but how these factors affect the models’ language perception is unclear. |
| Approach: | They compare the self-attention of several existing large language models in different sizes to assess the effect of scaling and instruction tuning on language perception. |
| Outcome: | The proposed models are closer to non-native speakers than native speakers in attention, suggesting a sub-optimal language perception of all models. |
Sparsity-Accelerated Training for Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) have demonstrated proficiency across various NLP tasks but often require additional training, such as continual pre-training and supervised fine-tuning. |
| Approach: | They propose to leverage sparsity in pre-trained LLMs to accelerate training by disregarding computations for unimportant neurons. |
| Outcome: | The proposed framework achieves comparable or superior performance to standard training while significantly accelerating the process. |
Let’s Rectify Step by Step: Improving Aspect-based Sentiment Analysis with Diffusion Models (2024.lrec-main)
Copied to clipboard
| Challenge: | Empirical evaluations conducted on eight benchmark datasets underscore the compelling advantages offered by DiffusionABSA when compared against robust baseline models. |
| Approach: | They propose a diffusion model which extracts aspects step by step and learns a denoising process that progressively restores them in a reverse manner. |
| Outcome: | Empirical evaluations on eight benchmark datasets underscore the compelling advantages offered by DiffusionABSA when compared against robust baseline models. |
Autoregressive Multi-trait Essay Scoring via Reinforcement Learning with Scoring-aware Multiple Rewards (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing reinforcement learning (RL) applications in AES are limited to classification models despite associated performance degradation. |
| Approach: | They propose to integrate actual evaluation schemes into the training process by designing QWK-based rewards with a mean-squared error penalty for multi-trait AES. |
| Outcome: | The proposed scoring-aware multi-reward reinforcement learning integrates actual evaluation schemes into the training process. |
A Meta-Learning Perspective on Transformers for Causal Language Modeling (2024.findings-acl)
Copied to clipboard
| Challenge: | Mechanisms of the Transformer architecture for causal language modeling are not well understood. |
| Approach: | They propose a meta-learning view of the Transformer architecture when trained for a causal language modeling task by explicating an inner optimization process that may happen within the Transformer. |
| Outcome: | The proposed model is based on a self-attention mechanism and has been widely used in natural language processing, computer vision, and scientific discovery. |
Mapping Work Task Descriptions from German Job Ads on the O*NET Work Activities Ontology (2024.lrec-main)
Copied to clipboard
| Challenge: | a new method for mapping job tasks to labor market ontologies is proposed . a top configuration of the method achieved a notable performance improvement . |
| Approach: | They use ontological data with Multiple Negatives Ranking loss to extract job tasks from job postings . they integrate labeled job advertisement data into training to improve their mapping . |
| Outcome: | The proposed method improves on the German job ads and their ontology . it can be used to map job tasks to established labor market ontologies or taxonomies . |
AutoMixer: Checkpoint Artifacts as Automatic Data Mixers (2025.acl-long)
Copied to clipboard
| Challenge: | In language model training, it is difficult to obtain the right data mixtures for various tasks as the relationship between data and tasks is difficult. |
| Approach: | They propose to identify checkpoint models based on their respective capabilities and leverage them as data mixers by using their aggregated first-order influence approximation over source data. |
| Outcome: | The proposed framework shows significant improvements on eight reasoning benchmarks, with accuracy increases of up to 1.93%. |
Masked Path Modeling for Vision-and-Language Navigation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | A major challenge in vision-and-language navigation is the limited available training data, which hinders the models’ ability to generalize effectively. |
| Approach: | They propose a masked path modeling objective that pretrains an agent using self-collected data for subsequent navigation tasks. |
| Outcome: | The proposed model pretrains an agent using self-collected data for subsequent navigation tasks eliminating the need for external tools. |
KE-MHISTO: Towards a Multilingual Historical Knowledge Extraction Benchmark for Addressing the Long-Tail Problem (2025.findings-acl)
Copied to clipboard
Arianna Graciotti, Leonardo Piano, Nicolas Lazzari, Enrico Daga, Rocco Tripodi, Valentina Presutti, Livio Pompianu
| Challenge: | Large Language Models struggle when probed for long-tail knowledge due to the inherent sparsity of such data. |
| Approach: | They propose a multilingual benchmark for Entity Linking and Question Answering in the domain of historical music knowledge that provides broader coverage of long-tail knowledge. |
| Outcome: | The proposed model provides broader coverage of long-tail knowledge compared to existing models. |
Adaptive Preference Optimization with Uncertainty-aware Utility Anchor (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Offline preference optimization methods are efficient for large language models (LLMs) alignment. |
| Approach: | They propose an offline preference optimization framework that estimates uncertainties from preference data . the method enables training even in scenarios where the data is unpaired . |
| Outcome: | The proposed method enables training even in scenarios where the data is unpaired . |
Mitigating Catastrophic Forgetting in Large Language Models with Forgetting-aware Pruning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in large language models (LLMs) have shown impressive capabilities in various downstream tasks but typically face Catastrophic Forgetting (CF) during fine-tuning. |
| Approach: | They propose a pruning-based approach to balance CF and downstream task performance by integrating the ratio of the task vector to pre-trained model parameters into the pruning criteria. |
| Outcome: | The proposed pruning-based approach limits CF to just 0.25% while maintaining 99.67% accuracy on downstream tasks. |
DCR: Quantifying Data Contamination in LLMs Evaluation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) memorize evaluation data during training, inflating performance metrics and undermining genuine generalization assessment. |
| Approach: | They propose a framework to detect and quantify benchmark data contamination (BDC) by synthesizing contamination scores via a fuzzy inference system. |
| Outcome: | The proposed framework detects and quantifies BDC risk across semantic, informational, data, and label levels. |
Learning to Insert [PAUSE] Tokens for Better Reasoning (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing studies have explored incorporating special-purpose tokens into the training process to enhance reasoning capabilities. |
| Approach: | They propose a method for inserting dummy tokens consecutively just before reasoning steps to increase model effectiveness. |
| Outcome: | The proposed method outperforms fine-tuning and previous token insertion methods on multiple datasets and models. |
Instability in Downstream Task Performance During LLM Pretraining (2025.findings-emnlp)
Copied to clipboard
| Challenge: | a study of large language models shows that task scores fluctuate throughout training . |
| Approach: | They empirically analyze the stability of downstream task performance in an LLM . |
| Outcome: | The proposed methods improve performance stability without changes to the training procedure. |
SynTOD: Augmented Response Synthesis for Robust End-to-End Task-Oriented Dialogue System (2024.lrec-main)
Copied to clipboard
| Challenge: | Task-oriented dialogue systems focus on training multiple tasks such as language understanding, tracking states, and generating appropriate responses to help users achieve their specific goals. |
| Approach: | They exploit the ability of pre-trained models to provide synthesis responses for fine-tuning end-to-end TOD systems. |
| Outcome: | The proposed model outperforms baseline models on multiwoz datasets and is available for further exploitation. |
Dynamic Jointly Batch Selection for Data Efficient Machine Translation Fine-Tuning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Data quality and effective selection are key to improving machine translation performance . study focuses on fine-tuning models using a batch selection strategy . |
| Approach: | They propose a data selection methodology for fine-tuning machine translation systems that leverages the synergy between a learner model and a pre-trained reference model to enhance overall training effectiveness. |
| Outcome: | The proposed method improves training efficiency by up to fivefold compared to baseline methods. |
Target-Adaptive Consistency Enhanced Prompt-Tuning for Multi-Domain Stance Detection (2024.lrec-main)
Copied to clipboard
| Challenge: | Stance detection is a fundamental task in natural language processing, but it is challenging due to diverse expressions and topics related to the targets from multiple domains. |
| Approach: | They propose a prompt-tuning method that incorporates target knowledge and prior knowledge to construct target-adaptive verbalizers for diverse domains. |
| Outcome: | The proposed method outperforms the state-of-the-art methods on nine stance detection datasets from multiple domains. |
DM-Codec: Distilling Multimodal Representations for Speech Tokenization (2025.findings-emnlp)
Copied to clipboard
Md Mubtasim Ahasan, Md Fahim, Tasnim Mohiuddin, Akmmahbubur Rahman, Aman Chadha, Tariq Iqbal, M Ashraful Amin, Md Mofijul Islam, Amin Ahsan Ali
| Challenge: | Existing speech tokenization models lack contextual representations for speech synthesis . absence of contextual representation results in elevated WER and WIL scores . |
| Approach: | They propose a language model-guided distillation method that incorporates contextual information into a comprehensive speech tokenizer. |
| Outcome: | The proposed method outperforms state-of-the-art tokenization models in reducing WER and WIL scores. |
HyperAdaLoRA: Accelerating LoRA Rank Allocation During Training via Hypernetworks without Sacrificing Performance (2026.findings-acl)
Copied to clipboard
Hao Zhang, Zhenjia Li, Yifan Gao, Xi Xiao, Heng Zhang, Shuyang Zhang, null Xiaoxincc, Bo Huang, Yuhang Wu, Tianyang Wang, Hao Xu
| Challenge: | Low-Rank Adaptation (LoRA) assumes a uniform rank r for each incremental matrix, not accounting for the varying significance of weight matrices across modules and layers. |
| Approach: | They propose a framework that allows for faster convergence of low-rank adaptive models . they use a hypernetwork to prune the outputs of the hypernetworks to generate parameters . |
| Outcome: | The proposed framework accelerates convergence of AdaLoRA by leveraging a hypernetwork. |